Conceptual Model Validation
Here we outline a workflow with algorithmic support in R for comparing our conceptual model to the existing literature corpus, the government policy documents and a qualitative assessment.
We apply this to systematically evaluate whether each component of our conceptual model is supported, refuted, missing from the literature and/or policy documents. We then synthesize this into a shorter, more justified and adjusted version of the model.
We build a semi-automated evidence synthesis framework that treats our documents like a team of virtual reviewers. The core idea is to define the core elements from your simple conceptual model (e.g. canteen regulation, staff training, food safety, nutrition education, etc.), search and summarize what each document says about each element, score the strength and direction of evidence per element, compare and synthesize to get a high-level view of agreement, gaps, and inconsistencies.
Decision framing
As a first step we framed the decision of implementing school meal policy. We determined our general understanding of the decision problem in the form of a simple causal model representing the important costs benefits and risks that policy will have at the school level.
Literure and policy review
we performed a review of policy and literature documents. Here we load the data resulting from the literature and policy review.
# Load .txt files
literature_text <- tolower(readLines("text_Review/Text_Lit_Review.txt"))
policy_text <- tolower(readLines("text_Review/Text_Policy_Review_Eng.txt"))
bib_policy <- read.bib("bib/school_meal_policy.bib")
bib_texts_policy <- tolower(sapply(bib_policy, function(x) {
paste(
x$title,
x$abstract,
if (!is.null(x$annote)) paste(x$annote, collapse = " ") else "",
collapse = " "
)
}))
# Load BibTeX literature entries as plain text
bib_literature <- read.bib("bib/school_meal_literature.bib")
# Load BibTeX literature entries as plain text
bib_texts_literature <- tolower(sapply(bib_literature, function(x) {
paste(
x$title,
x$abstract,
if (!is.null(x$annote)) paste(x$annote, collapse = " ") else "",
collapse = " "
)
}))Algorithmic support
To compare our conceptual model to the existing literature, the government policy documents and a qualitative assessment we first the corpus search for key terms.
food_tech_training
Infrastructure, equipment, and capacity-building components needed for food preparation and delivery in schools.
food_tech_training = c(
"kitchen equipment", "cooking equipment", "commercial kitchen",
"school meal preparation", "technical food support",
"canteen operations", "food service logistics",
"kitchen operations", "meal service infrastructure",
"food production capacity", "food storage infrastructure",
"kitchen staff training", "food preparation training",
"school catering system", "school meal delivery",
"bulk cooking systems", "school kitchen upgrade",
"food handling facility", "school feeding infrastructure", "trained and educated on proper nutrition"
)teaching_resources
Time, personnel, and instructional capacity needed for teachers and staff to implement nutrition interventions.
teaching_resources = c(
"teaching workload", "teacher time", "instructional time",
"curricular demands", "lesson planning constraints",
"staffing capacity", "teacher availability",
"instructional burden", "curriculum overcrowding",
"limited teaching time", "staffing shortage",
"time allocation", "teaching schedule",
"instructional capacity", "non-teaching responsibilities",
"classroom time pressure", "teacher deployment",
"resource constraints for instruction"
)staff_skill
The pedagogical and technical competencies of school staff related to nutrition, health, and food safety.
staff_skill = c(
"staff nutrition knowledge", "teacher nutrition competency",
"school health educators", "teacher skills",
"staff training", "training effectiveness",
"professional development for teachers",
"teacher capacity building", "staff development programs",
"food literacy training", "capacity development",
"in-service training", "teacher competence",
"educator readiness", "pedagogical support for nutrition",
"staff preparation", "training coverage",
"staff instructional skills", "training quality",
"nutrition educator training"
)intervention_cost
Economic factors influencing the feasibility, sustainability, and scalability of school-based nutrition interventions, including direct costs and systemic financial constraints.
intervention_cost = c(
"intervention cost", "program cost", "implementation cost",
"financial feasibility", "budget constraint", "school finance",
"economic barrier", "affordability", "funding shortfall",
"resource allocation", "cost-effectiveness", "cost-benefit",
"financial sustainability", "school budget", "budget limitation",
"fiscal capacity", "operational cost", "nutrition program funding",
"financial barrier", "economic feasibility", "budget planning",
"cost per child", "nutrition intervention expenses"
)canteen_change
Policy-driven or structural reforms to the school food environment, particularly within canteen services, aimed at improving food quality, nutritional adequacy, and student access to healthy meals.
canteen_change = c(
"canteen reform", "school meal change", "menu revision",
"canteen intervention", "food service reform",
"school food environment", "healthy food provision",
"nutrition-sensitive canteen", "school food policy",
"canteen nutrition standards", "meal plan reform",
"food service update", "school food regulation",
"canteen policy enforcement", "nutritional canteen shift",
"menu redesign", "food procurement reform",
"canteen-based intervention", "school food service management"
)nutrition_lessons
Educational initiatives and curricular strategies that provide students with knowledge, skills, and attitudes related to nutrition, diet, and healthy lifestyle practices.
nutrition_lessons = c(
"nutrition education", "health class", "food literacy lessons",
"nutrition curriculum", "classroom food education",
"school-based nutrition instruction", "dietary education",
"nutrition teaching materials", "health promotion teaching",
"lesson on healthy eating", "curriculum-integrated nutrition",
"food education", "nutrition behavior curriculum",
"classroom nutrition program", "healthy eating curriculum",
"nutrition awareness education", "school nutrition module",
"teacher-led nutrition instruction", "classroom wellness lesson"
)food_ads
Advertising and commercial promotion of unhealthy foods in the school environment or within children’s media ecosystems that influence eating behaviors and preferences.
food_ads = c(
"food marketing", "junk food advertising", "unhealthy food ads",
"child-targeted food marketing", "ads near school",
"commercial food promotion", "branded snack promotion",
"advertising to children", "point of sale marketing",
"school food advertising", "child-directed advertising",
"food and beverage marketing", "media food influence",
"food advertisements", "billboard marketing",
"TV food commercials", "digital food marketing",
"sugar-sweetened beverage marketing", "fast food advertising",
"packaging marketing to kids"
)off_campus
The external food environment accessible to students outside of school grounds, including vendors, markets, and informal sources that often provide low-nutrient, energy-dense foods.
off_campus = c(
"off-campus eating", "external food", "street food vendors",
"gate food", "junk food near school", "mobile food vendors",
"non-canteen food", "external food purchases",
"school gate vendors", "food from outside school",
"unregulated food sales", "open food access",
"snack carts near school", "neighborhood food environment",
"informal food sources", "out-of-school food access",
"off-premises food", "community vendors near school"
)on_campus
Food consumption and dietary behaviors that occur within school premises, including those influenced by school meals, canteen offerings, and in-school food policies.
on_campus = c(
"school food consumption", "on-site meal intake",
"canteen food intake", "in-school dietary habits",
"school-based eating", "healthy eating in school",
"school meal participation", "in-school food intake",
"on-campus nutrition", "eating during school hours",
"classroom snack practices", "canteen-based eating",
"student food behavior in school", "regulated food intake",
"school food environment behavior", "food consumption at school"
)student_learning
Academic and cognitive outcomes linked to nutrition and school health environments, including concentration, performance, and educational attainment.
student_learning = c(
"learning outcomes", "academic performance", "school achievement",
"cognitive benefit", "cognitive development", "classroom concentration",
"student attention span", "education impact",
"student knowledge gain", "test scores",
"reading comprehension", "numeracy outcomes",
"academic success", "educational attainment",
"classroom engagement", "learning readiness",
"mental focus", "academic participation",
"nutrition and learning", "school-based academic improvement"
)student_health
Physical health outcomes affected by school food environments, including growth, weight status, illness prevalence, and long-term well-being.
student_health = c(
"child health", "student well-being", "healthy weight",
"BMI improvement", "nutritional status",
"diet-related health", "physical development",
"illness reduction", "health outcome", "childhood obesity",
"malnutrition", "nutrition-related disease",
"student physical health", "diet quality outcome",
"chronic disease prevention", "health behavior",
"public health outcome", "body mass index",
"school nutrition outcomes", "growth monitoring"
)search_terms <- list(
food_tech_training = food_tech_training,
teaching_resources = teaching_resources,
staff_skill = staff_skill,
intervention_cost = intervention_cost,
canteen_change = canteen_change,
nutrition_lessons = nutrition_lessons,
food_ads = food_ads,
off_campus = off_campus,
on_campus = on_campus,
student_learning = student_learning,
student_health = student_health
)Check Text Hits
check_hits <- function(search_terms, text) {
any(sapply(search_terms, function(t) grepl(t, text, ignore.case = TRUE)))
}
comparison_matrix <- data.frame(
bib_data_policy = sapply(search_terms, check_hits, text = paste(bib_texts_policy, collapse = " ")),
bib_data_lit = sapply(search_terms, check_hits, text = paste(bib_texts_literature, collapse = " ")),
gov_review = sapply(search_terms, check_hits, text = paste(policy_text, collapse = " ")),
lit_review = sapply(search_terms, check_hits, text = paste(literature_text, collapse = " "))
)Summary Table full model
# Add numeric sum and evidence strength labels
comparison_matrix$support_score <- rowSums(comparison_matrix[ , c("bib_data_policy", "gov_review", "bib_data_lit", "lit_review")])
comparison_matrix$evidence_strength <- cut(
comparison_matrix$support_score,
breaks = c(-1, 0.1, 1.3, 3.1, 4.1),
labels = c("", "Weak", "Moderate", "Strong"),
right = TRUE
)
# Clean row names
rownames(comparison_matrix) <- gsub("_", " ", rownames(comparison_matrix))
rownames(comparison_matrix) <- stringr::str_to_title(rownames(comparison_matrix))
# Clean column names
colnames(comparison_matrix) <- gsub("_", " ", colnames(comparison_matrix))
colnames(comparison_matrix) <- stringr::str_to_title(colnames(comparison_matrix))
knitr::kable(
comparison_matrix[, !(names(comparison_matrix) %in% "Support Score")],
caption = "Model Component Evidence Presence"
)| Bib Data Policy | Bib Data Lit | Gov Review | Lit Review | Evidence Strength | |
|---|---|---|---|---|---|
| Food Tech Training | TRUE | FALSE | FALSE | FALSE | Weak |
| Teaching Resources | FALSE | TRUE | FALSE | FALSE | Weak |
| Staff Skill | FALSE | TRUE | FALSE | FALSE | Weak |
| Intervention Cost | FALSE | TRUE | FALSE | TRUE | Moderate |
| Canteen Change | FALSE | TRUE | FALSE | TRUE | Moderate |
| Nutrition Lessons | TRUE | TRUE | FALSE | TRUE | Moderate |
| Food Ads | FALSE | TRUE | FALSE | FALSE | Weak |
| Off Campus | FALSE | FALSE | FALSE | FALSE | |
| On Campus | FALSE | FALSE | FALSE | FALSE | |
| Student Learning | TRUE | TRUE | FALSE | TRUE | Moderate |
| Student Health | TRUE | TRUE | TRUE | TRUE | Strong |
Count citations
source("functions/get_hits_by_bib.R")
literature_hits <- get_hits_by_bib(search_terms, bib_texts_literature)
policy_hits <- get_hits_by_bib(search_terms, bib_texts_policy)
summary_hits <- data.frame(
Literature_Count = sapply(literature_hits, length),
Policy_Count = sapply(policy_hits, length)
)
knitr::kable(summary_hits, caption = "Evidence Hits per Concept")| Literature_Count | Policy_Count | |
|---|---|---|
| food_tech_training | 0 | 1 |
| teaching_resources | 1 | 0 |
| staff_skill | 1 | 0 |
| intervention_cost | 1 | 0 |
| canteen_change | 6 | 0 |
| nutrition_lessons | 11 | 4 |
| food_ads | 1 | 0 |
| off_campus | 0 | 0 |
| on_campus | 0 | 0 |
| student_learning | 2 | 1 |
| student_health | 10 | 5 |
Policy support for our model
In policy documents we found supporting evidence for the variables we expressed in our model:
1 referencing food tech training.
Minister (2019)0 referencing teaching resources.
0 referencing staff skill.
0 referencing intervention cost.
0 referencing canteen change.
4 referencing nutrition lessons.
Education and Training (2022a), Minister (2021), Minister (2022), Minister (2016)0 referencing food ads.
0 referencing off-campus food.
0 referencing on-campus food.
1 referencing student learning.
Assembly (2019)5 referencing student health.
Minister (2022), Education and Training (2022b), Minister (2016), Minister (2019), Health (2019)
Literature support for our model
In the literature we found supporting evidence for the variables we expressed in our model:
0 referencing food tech training.
1 referencing teaching resources.
Wu et al. (2015)1 referencing staff skill.
Ouda et al. (2019)1 referencing intervention cost.
Parnham et al. (2022)6 referencing canteen change.
Wu et al. (2015), Grigsby-Duffy et al. (2022), S. Pongutta et al. (2023), S. Pongutta et al. (2022), Castellari and Berning (2016), Martinelli et al. (2023)11 referencing nutrition lessons.
Steyn et al. (2015), S. Pongutta et al. (2023), Taylor et al. (2011), S. Pongutta et al. (2022), Aroesty et al. (2018), Trung Le (2012), Hockamp et al. (2024), Liou et al. (2015), Walton, Signal, and Thomson (2013), Ohri-Vachaspati et al. (2016), Woo (2015)1 referencing food ads.
Walton, Signal, and Thomson (2013)0 referencing off-campus food.
0 referencing on-campus food.
2 referencing student learning.
Ouda et al. (2019), Aroesty et al. (2018)10 referencing student health.
Wu et al. (2015), Parnham et al. (2022), S. Pongutta et al. (2023), Taylor et al. (2011), S. Pongutta et al. (2022), Aroesty et al. (2018), Trung Le (2012), Liou et al. (2015), Walton, Signal, and Thomson (2013), Martinelli et al. (2023)
Model update and Monte Carlo Simulation
We take these findings into account and also talk to experts to update the model.
We ran the full model using Monte Carlo
simulation to estimate the likely outcomes of different school nutrition
policy configurations.
The outputs include both health cost savings (e.g. avoided diagnoses and treatments) and economic return (net present value compared to the baseline). These are plotted below to visualize the decision space.
library(ggplot2)
library(decisionSupport)
source("functions/School_Policy_Model.R")
# Load inputs and run simulation
input_data <- estimate_read_csv("data/inputs_school_policy.csv")
numberOfModelRuns <- 10000
set.seed(84)
simulation_result <- mcSimulation(
estimate = input_data,
model_function = school_policy_function,
numberOfModelRuns = numberOfModelRuns,
functionSyntax = "plainNames"
)## Warning: Variable: n_reduce_disease_treatment distribution: posnorm
## Calculated value of 5%-quantile: 0.0176119007818566
## Target value of 5%-quantile: 0.01
## Calculated cumulative probability at value 0.01 : 0.0352941176470588
## Target cumulative probability at value 0.01 : 0.05
## Mean scaled difference: 0.2941176## Warning in paramtnormci_numeric(p = p, ci = ci, lowerTrunc = lowerTrunc, : Calculated value of 5%-quantile: 0.0176119007818566
## Target value of 5%-quantile: 0.01
## Calculated cumulative probability at value 0.01 : 0.0352941176470588
## Target cumulative probability at value 0.01 : 0.05
## Mean scaled difference: 0.2941176## Warning: Variable: training_costs_nutrition_annual distribution: posnorm
## Calculated value of 5%-quantile: 0.263367695499241
## Target value of 5%-quantile: 0.01
## Calculated cumulative probability at value 0.01 : 0.00195944406470722
## Target cumulative probability at value 0.01 : 0.05
## Mean scaled difference: 0.9608111## Warning in paramtnormci_numeric(p = p, ci = ci, lowerTrunc = lowerTrunc, : Calculated value of 5%-quantile: 0.263367695499241
## Target value of 5%-quantile: 0.01
## Calculated cumulative probability at value 0.01 : 0.00195944406470722
## Target cumulative probability at value 0.01 : 0.05
## Mean scaled difference: 0.9608111## Warning: Variable: training_costs_physical_activity_annual distribution: posnorm
## Calculated value of 5%-quantile: 0.17466915995397
## Target value of 5%-quantile: 0.01
## Calculated cumulative probability at value 0.01 : 0.0032051282051282
## Target cumulative probability at value 0.01 : 0.05
## Mean scaled difference: 0.9358974## Warning in paramtnormci_numeric(p = p, ci = ci, lowerTrunc = lowerTrunc, : Calculated value of 5%-quantile: 0.17466915995397
## Target value of 5%-quantile: 0.01
## Calculated cumulative probability at value 0.01 : 0.0032051282051282
## Target cumulative probability at value 0.01 : 0.05
## Mean scaled difference: 0.9358974## Warning: Variable: monitoring_canteen_cost distribution: posnorm
## Calculated value of 5%-quantile: 0.088115775763901
## Target value of 5%-quantile: 0.05
## Calculated cumulative probability at value 0.05 : 0.0289473684210526
## Target cumulative probability at value 0.05 : 0.05
## Mean scaled difference: 0.4210526## Warning in paramtnormci_numeric(p = p, ci = ci, lowerTrunc = lowerTrunc, : Calculated value of 5%-quantile: 0.088115775763901
## Target value of 5%-quantile: 0.05
## Calculated cumulative probability at value 0.05 : 0.0289473684210526
## Target cumulative probability at value 0.05 : 0.05
## Mean scaled difference: 0.4210526# Scatter plot of health vs. economic return
ggplot(simulation_result$y, aes(x = decision_value, y = health_cost_savings)) +
geom_point(alpha = 0.3) +
labs(
title = "Simulated Policy Outcomes",
x = "Economic Return (NPV: Policy – No Policy) [million VND]",
y = "Health Cost Savings [million VND]"
) +
theme_minimal()
The model was run 10^{4} times to capture uncertainty across a wide range of possible scenarios.
# Assume sim_result is the mcSimulation result
sim_data <- data.frame(simulation_result$x, simulation_result$y[1:2]) # Extract all output values
# Define simple Pareto filter
pareto_filter <- function(df, obj1, obj2) {
pareto <- rep(TRUE, nrow(df))
for (i in 1:nrow(df)) {
pareto[i] <- !any(
df[[obj1]] > df[[obj1]][i] &
df[[obj2]] >= df[[obj2]][i] |
df[[obj1]] >= df[[obj1]][i] &
df[[obj2]] > df[[obj2]][i]
)
}
return(df[pareto, ])
}
# Apply to individual simulation runs
pareto_front <- pareto_filter(sim_data, "decision_value", "health_cost_savings")